IMAGE-CAPTURING APPARATUS 



BACKGROUND OF THE INVENTION 

1. Field of the Invention 

The present invention relates to an image-capturing 
apparatus achieved by using a solid-state image-capturing 
element . 

2. Description of the Related Art 

Solid-state image-capturing apparatuses that improve 
the sensitivity by adding together electrical charges stored 
at adjacent pixels along the horizontal direction at a CCD 
solid-state image-capturing element and also minimize the 
extent of degradation of the horizontal resolution 
attributable to the addition include the one disclosed in 
Japanese Laid Open Patent Publication No. H 06-209434. 

Image signals are processed as described below in the 
image-capturing apparatus in the related art mentioned above. 
A reset pulse is applied to a reset gate of the CCD solid-state 
image-capturing element through intermittent drive over 
2^pixel cycles and thus, the electrical charges stored at 
pixels adjacent to each other along the horizontal direction 
are mixed together to realize a sensitivity level twice as 
high. In addition, by adopting a combination of phase control 
of the reset pulse and interpolation processing of signals 
achieved based upon vertical correlation, the extent to which 



the horizontal resolution is lowered is reduced to 3/4. 
However, it is inevitable that the resolution still becomes 
poorer . 



5 SUMMARY OF THE INVENTION 

According to the present invention, outputs of pixels 
located around a given pixel at an image-capturing unit are 
added to the output of that pixel before executing image 
processing on image signals output from the image-capturing 

10 unit having a plurality of pixels disposed two-dimensionally . 
An addition pattern with which pixel outputs are added together 
is determined in correspondence to the specific type of image 
processing to be executed and the pixel outputs constituting 
the image signals output from the image-capturing unit are 

15 added in conformance to the addition pattern to generate an 
image. Then, the image processing is executed on the image 
resulting from the addition and the image having undergone 
the image processing is output. 

Instead of determining the addition pattern with which 

20 'the pixel outputs are added together in correspondence to the 
type of image processing, the addition pattern may be 
determined in correspondence to a specific vehicular behavior . 
Alternatively, the pixel output addition pattern may be 
determined in correspondence to the extent of vibration that 

25 the image-capturing apparatus experiences . 
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BRIEF DESCRIPTION OF THE DRAWINGS 
FIG . 1 shows the structure adopted in the image-capturing 
apparatus achieved in a first embodiment; 
5 FIG. 2 shows the results of image processing executed 

by adding pixel outputs along the longitudinal direction; 

FIG. 3 shows the results of image processing executed 
by adding pixel outputs along the lateral direction; 

FIG. 4 shows the results of image processing executed 
10 by adding pixel outputs continuous in a pattern; 

FIG. 5 shows the results of image processing executed 
by adding pixel outputs continuous in a "\" pattern; 

FIG. 6 presents an example of a variation of the 
image-capturing apparatus achieved in the first embodiment; 
15 FIG. 7 presents another example of a variation of the 

image-capturing apparatus achieved in the first embodiment; 
and 

FIG. 8 shows the structure adopted in the image-capturing 
apparatus achieved in a second embodiment. 

20 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 
( First Embodiment ) 

FIG. 1 shows the structure adopted in an image-capturing 
apparatus achieved in the first embodiment. The 
25 image-capturing apparatus includes an image-capturing 
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element 1, an adder 2, an A/D converter 3, an processing device 
4 and an addition pattern generating device 5. 

The image-capturing element 1 is a semiconductor device 
adopting a CMOS structure which includes a plurality of light 
5 receiving elements disposed two-dimensionally and a read 
circuit that reads output voltages from the individual light 
receiving elements. The image-capturing element 1 outputs 
the output voltages from light receiving elements specified 
with read address signals as output voltages of the 

10 image-capturing element 1. 

The adder 2 adds the output signals from light receiving 
elements at addresses specified by the addition pattern 
generating device 5 which is to be detailed later until a reset 
signal is output from the addition pattern generating device 

15 5. In other words, the adder 2 generates a new image by adding 
the outputs of pixels present around a given pixel at the 
image-capturing element 1 to the output from the pixel. 

During the add processing executed by the adder 2, the 
pixel outputs may be added together by weighting the outputs 

20 in correspondence to the individual light receiving elements 
instead of simply adding them together. In such a case, the 
addition pattern generating device 5 may output weighting 
elements, or if weighting elements are fixed, the weighting 
elements may be automatically generated at the adder 2. 

25 The A/D converter 3 executes analog-digital conversion 
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so as to enable digital processing of the output from the adder 
2. The A/D converter 3 is achieved by using a device and 
circuits known in the related art. 

The processing device 4 executes image processing on 
5 the image-capturing signals having undergone the digital 
conversion to detect a vehicle present ahead, to detect the 
road conditions and the like. Such detections can be executed 
by adopting a method known in the related art. The processing 
device 4 also outputs information indicating the type of image 

10 processing to be executed to the addition pattern generating 
device 5. For instance, when executing longitudinal edge 
detection (vertical edge detection) through longitudinal 
Sobel filter arithmetic processing on 3 x 3 pixel data to detect 
a white line drawn on a road, a signal indicating that image 

15 processing for longitudinal edge detection is to be executed 
(hereafter referred to as an image processing type signal) 
is output to the addition pattern generating device 5. 

The addition pattern generating device 5, to which the 
image processing type signal is input from the processing 

20 device 4 , generates a combination pattern or an addition 
pattern with which pixels are to be combined for the addition 
of the outputs thereof in correspondence to the processing 
type and outputs address signals specifying light receiving 
elements in correspondence to the addition pattern to the 

25 image-capturing element 1 . In addition, the addition pattern 
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generating device 5 outputs a reset signal to the adder 2 to 
initialize the adder 2. 

The combination pattern with which pixels are combined 
for the addition in correspondence to the type of image 
5 processing should be regarded as a pattern that will yield 
optimal results for a specific processing purpose, e.g., a 
pattern with which the outputs from three light receiving 
elements continuous along the lateral direction are added 
together or a pattern with which the outputs of every second 

10 light receiving element along the longitudinal direction for 
a total of four light receiving elements are added together. 

FIGS. 2 to 5 show the results of image processing 
executed by using various sum image signals obtained in the 
image-capturing apparatus in the first embodiment. In FIGS. 

15 2-5, images 100, 200, 300 and 400 show the original image 
captured by the image-capturing element 1, images 110, 210, 
310 and 410 show a new image generated by adding the outputs 
of pixels present around a given pixel to the output of the 
pixel through a specific addition pattern at the adder 2 and 

20 images 120, 220, 320 and 420 show the results of a specific 
edge detection executed by the processing device 4 by using 
the new image. 

FIG. 2 shows the results of image processing executed 
by using the sums of outputs from pixels continuous along the 

25 longitudinal direction (vertical direction) . By adding 
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together the outputs of pixels continuous along the 
longitudinal direction, a bright image can be obtained without 
lowering the resolution along the lateral direction 
(horizontal direction) . For this reason, a bright and 
5 high-contrast image can be obtained even in an environment 
with a low level of illumination, e.g., night time . The outputs 
f rompixels continuous along the longitudinal direction should 
be added together when, for instance, the processing device 
4 is to execute image processing such as pedestrian detection 

10 in which a longitudinal edge component (a vertical edge 
component) of a target object needs to be extracted accurately. 

FIG. 3 shows the results of image processing executed 
by using the sums of outputs from pixels continuous along the 
lateral direction. By adding together the outputs of pixels 

15 continuous along the lateral direction, a bright image can 
be obtained without lowering the resolution along the 
longitudinal direction. For this reason, a bright and high 
contrast image can be obtained even in an environment with 
a low level of illumination, e.g. , night time. For instance, 

20 when detecting a preceding vehicle or a succeeding vehicle 
processing is executed to extract a lateral edge component 
(a horizontal edge component) of the target object . As a result, 
a highly accurate lateral edge extraction is achieved even 
when the level of the illumination is low without reducing 

25 the resolution along the longitudinal direction by adding 
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together the outputs from pixels continuous along the lateral 
direction. In addition, by adding together the outputs of 
pixels continuous along the lateral direction, an image can 
be generated having a lateral edge component more pronounced 
than the longitudinal edge component. 

By switching to an appropriate addition pattern in 
correspondence to the type of image processing executed by 
the processing device 4 to extract necessary information from 
the image data, the target object can be detected even when 
the level of illumination is low, without compromising the 
level of accuracy. 

In each of the examples presented in FIGS. 2 and 3, 
a bright image is obtained without reducing the information 
corresponding to the vertical edges or the horizontal edges 
by adding the outputs of pixels continuous along the x vertical 
direction or the horizontal direction in a one-dimensional 
pattern. However, the advantage of the invention can be 
further enhanced by expanding the addition pattern over a 
two-dimensional plane . 

FIG. 4 shows the results of image processing executed 
on image signals obtained by adding the outputs of pixels 
continuous in a "V pattern. By adding together the outputs 
of pixels continuous in the pattern, a bright image in 

which the characteristics of the "-i" pattern are emphasized 
is generated. This addition pattern may be adopted when, for 



8 



instance, the corners of a vehicle must be detected with a 
high degree of accuracy. 

FIG. 5 shows the results of image processing executed 
on image signals obtained by adding the outputs of pixels 
5 continuous in a "\" pattern. By adding together the outputs 
of pixels continuous in the f, \" pattern, a bright image in 
which the characteristics of the "\" pattern are emphasized 
is generated. This addition pattern may be adopted when, for 
instance, a white line drawn diagonally on a road must be 

10 detected with a high degree of accuracy. 

The addition patterns are not limited to the longitudinal 
pattern, the lateral pattern, the pattern and the "\" 

pattern shown in FIGS . 2 to 5 . Various other patterns including, 
for instance, a " + " pattern, a "/\" pattern and "A" pattern 

15 may be used. 

As described above, in the image-capturing apparatus 
achieved in the embodiment, the outputs of pixels present in 
the vicinity of a given pixel at the image-capturing element 
1 are added onto the output of the pixel. The addition is 

20 executed based upon the pixel output addition pattern 
determined in correspondence to the type of image processing 
to be executed. As a result, when the processing device 4 
performs image processing by using the image signals provided 
by the image-capturing apparatus, an edge detection can be 

25 executed in a desirable manner in an image with improved 
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sensitivity in a low illumination condition without lowering 
the resolution. 

In addition, by generating various addition patterns 
in correspondence to varying types of edges in the image to 
5 be detected through the image processing, the different types 
of edges, e.g., longitudinal edges, lateral edges, diagonal 
edges and a -.-shaped edge, can be detected with a high degree 
of accuracy. 

Furthermore, by generating the addition pattern in 

10 correspondence to a specific detection target type when 
detecting target objects such as pedestrians or vehicles 
through image processing, pedestrians or vehicles ahead of 
or behind the subject vehicle can be detected reliably. 

The addition pattern may be altered for each frame of 

15 image output by the image-capturing element 1. For instance, 
the addition pattern may be switched for each frame by 
alternating between the longitudinal pattern and the lateral 
pattern. In this case, the processing device 4 should execute 
longitudinal edge extraction processing and lateral edge 

20 extraction processing in combination, each in correspondence 
to one of the addition patterns. It is particularly 
advantageous to change the addition pattern for each frame 
when an image-capturing element with a high frame rate is used. 
Since the time lag between the longitudinal addition and the 

25 lateral addition is reduced when the two types of processing 
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are alternately executed repeatedly at a high frame rate, 
processing results equivalent to the results achieved by 
simultaneously extracting the longitudinal edges and the 
lateral edges with a high-sensitivity image are obtained. 
5 FIG. 6 presents another example of an image-capturing 

apparatus that adjusts the addition pattern in correspondence 
to the frame rate or the length of exposure time. In FIG. 
6, the same reference numerals are assigned to components 
identical to those in the image-capturing apparatus shown in 

10 FIG. 1 to preclude the necessity for a repeated explanation 
thereof . Reference numeral 11 indicates a setting device that 
sets either a frame rate or a length of exposure time. 

When the processing device 4 executes image processing 
for lateral edge detection, for instance, the number of pixels 

15 continuous along the lateral direction, the outputs of which 
are to be added together, can be adjusted in correspondence 
to the frame rate or the length of exposure time set by the 
setting device 11. To rephrase this by referring to the 
addition pattern as a lateral addition pattern, the number 

20 of pixels the outputs of which are added in the lateral addition 
pattern is adjusted in correspondence to the frame rate or 
the length of exposure time. 

By executing the image processing with the image signals 
output from the image-capturing apparatus described above, 

25 the dynamic range can be expanded while achieving the required 
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level of sensitivity to minimize the extent of deterioration 
of resolution. 

FIG. 7 shows another example of an image-capturing 
apparatus that adjusts the addition pattern in correspondence 
5 to the extent of vibration the image-capturing apparatus 
experiences. In FIG. 7, the same reference numerals are 
assigned to components identical to those in the 
image-capturing apparatus shown in FIG. 6 to preclude the 
necessity for a repeated explanation thereof. Reference 
10 numeral 21 indicates a vibration detection sensor that detects 
movement (vibration) of the opt ical axis of the image-capturing 
apparatus . 

As the optical axis of a photographic optical system 
which forms a subject image on the image-capturing element 

15 1 shifts, the image becomes blurred or smeared. In such a 
case, the vibration detection sensor 21 detects the movement 
(vibration) of the optical axis of the image-capturing 
apparatus. Then, an addition pattern is generated so as to 
add together the outputs of pixels disposed continuously along 

20 a direction matching the detected direction, i.e., the 
direction along which the image blur or smear has occurred. 

By executing the image processing with the image signals 
output from such an image-capturing apparatus, the detection 
target image can be extracted with a high degree of sensitivity 

25 without lowering the resolution even when the optical axis 
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of the image-capturing apparatus has shifted. 
( Second Embodiment ) 

An image-capturing apparatus achieved in the second 
embodiment, which detects vehicular behavior and switches the 
5 pixel addition pattern in correspondence to the vehicular 
behavior, is explained in reference to FIG. 8. The same 
reference numerals are assigned to components and devices 
identical to those shown in FIG. 1 and the following explanation 
focuses on the differences. 
10 An A/D converter 3A executes analog - digital conversion 

to enable digital processing of the output of the 
image-capturing element 1. In this embodiment, the number 
of quantization bits is defined as follows. 

(number of A/D converter output bits) = (number of adder output 
15 bits) + log 2 (number of pixels used for addition) 

This makes it possible to achieve a high level of resolution 
at which the image processing can be executed without losing 
any information even under a condition of low illumination. 

An adder 2A executes digital processing to add the output 
20 signals from light receiving elements at addresses specified 
by the addition pattern generating device 5 until a reset signal 
is output from the addition pattern generating device 5. 

In the second embodiment, signals output from the 
image-capturing element 1 immediately undergo the A/D 
25 conversion and are then added together through the digital 
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processing. However, the output signals from the 

image-capturing element 1 may insteadbe added together through 
analog processing and the results of the addition may then 
undergo A/D conversion as in the first embodiment. 
5 A vehicular behavior measuring device 6 measures the 

vehicular behavior. A specific addition pattern type is 
determined based upon, for instance, information provided by 
a yaw rate sensor, which measures the vehicle motion along 
the yawing direction effected through steering and information 
10 provided by a vehicle height sensor which measures vertical 
movement of the vehicle. The addition pattern type thus 
determined is output to the addition pattern generating device 
5. 

When a pronounced vertical motion occurs at the vehicle, 
15 a pattern with which outputs are added along the longitudinal 
direction is indicated. When a pronounced vertical motion 
is occurring at the vehicle, the image becomes blurred along 
the longitudinal direction resulting in a lowered resolution 
along the longitudinal direction. For this reason, the 
20 resolution does not become any lower if the signals output 
from the pixels disposed along the longitudinal direction are 
added together . When the yaw rate of the vehicle is significant, 
on the other hand, a pattern with which outputs are added 
together along the lateral direction is indicated. When the 
25 yaw rate of the vehicle is high, the image becomes blurred 
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along the lateral direction resulting in a lowered resolution 
along the lateral direction. For this reason, the resolution 
does not become any lower if the signals output from the pixels 
disposed along the lateral direction are added together. 
5 Since the results of the image processing executed in 

the second embodiment are similar to the image processing 
results achieved in the first embodiment shown in FIGS. 2 
to 5, their explanation is omitted. 

The image-capturing apparatus achieved in the second 
10 embodiment detects the vehicular behavior and generates the 
addition pattern corresponding to the vehicular behavior. 
Thus, even when the vehicle moves up and down or to the left /right 
while traveling, the sensitivity under low illumination 
conditions can be improved while minimizing the extent of the 
15 deterioration of the resolution. 

In the second embodiment, the addition pattern is 
adjusted based upon the vehicular behavior. This addition 
pattern adjustment may be adopted in conjunction with the 
addition pattern adjustment based upon the type of image 
20 processing having been explained in reference to the first 
embodiment and the examples of variations. 

The image-capturing apparatus according to the present 
invention is used in a vehicle both in the first embodiment 
and in the second embodiment described above. However, the 
25 present invention may be adopted in all types of applications 
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other than automotive applications, as well. For instance, 
it may be adopted in an image-capturing apparatus for a robot 
that detects the position of an object. 

It is to be noted that the individual components do not 
5 need to adopt the specific structures described above as long 
as the functions characterizing the present invention are not 
compromised . 

The disclosures of the following priority applications 
are incorporated herein by reference: 
10 Japanese Patent Application No. 2003-142536 filed May 20, 
2003 

Japanese Patent Application No. 2004-072530 filed March 15, 
2004 
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